Add Elastic Agent metadata to the org-data command#347
Open
Conversation
Updates the organization data model and integration logic to include stable Elastic Agent identification in all generated documents. This ensures that synthetic data correctly mimics real-world ECS-compliant logs and supports security features that rely on agent-based correlation. Key changes: - Adds `elasticAgentId` to Device and Host models, and `centralAgent` to the Organization model. - Implements `buildCentralAgent`, `buildLocalAgent`, and `buildServerAgent` helpers in `BaseIntegration`. - Updates all existing integrations to include the `agent` field in their documents. - Expands the Okta system integration with anomalous event generation to support testing for Post-Authentication Detection (PAD).
Contributor
There was a problem hiding this comment.
Pull request overview
This PR extends the org-data synthetic data generator to attach ECS agent.* metadata to all generated documents, using stable identifiers for local workstation agents, server agents, and a single central collector agent for SaaS/cloud integrations. It also expands the Okta System Logs integration with additional org-level and anomalous event generation to better exercise Post-Authentication Detection (PAD) scenarios.
Changes:
- Adds
elasticAgentIdtoDeviceandHost, pluscentralAgenttoOrganization, and generates these values in the org data generator. - Introduces
buildCentralAgent,buildLocalAgent, andbuildServerAgenthelpers (andAgentData) inBaseIntegration, then wiresagentinto integration documents across many integrations. - Enhances
okta_systemevent generation with org-level lifecycle/privilege/app assignment events and anomalous PAD-oriented patterns.
Reviewed changes
Copilot reviewed 54 out of 54 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| src/commands/org_data/types.ts | Adds agent-related fields/types (elasticAgentId, CentralAgent, Organization.centralAgent). |
| src/commands/org_data/org_data_generator.ts | Generates centralAgent, plus elasticAgentId for devices/hosts. |
| src/commands/org_data/integrations/base_integration.ts | Adds AgentData, ELASTIC_AGENT_VERSION, and agent builder helpers for central/local/server agents. |
| src/commands/org_data/integrations/endpoint_integration.ts | Switches to device.elasticAgentId, standardizes agent version, and adds hostname to agent object. |
| src/commands/org_data/integrations/system_integration.ts | Adds server agent metadata via buildServerAgent(host) in emitted docs. |
| src/commands/org_data/integrations/zscaler_zia_integration.ts | Adds local agent metadata to web/firewall docs (currently conditional). |
| src/commands/org_data/integrations/jamf_pro_integration.ts | Adds local agent metadata tied to workstation hostname. |
| src/commands/org_data/integrations/island_browser_integration.ts | Adds central agent to SaaS docs and local agent to device/audit docs. |
| src/commands/org_data/integrations/crowdstrike_integration.ts | Adds local/central agent metadata depending on Falcon event type and document. |
| src/commands/org_data/integrations/okta_integration.ts | Adds central agent to sync marker and entity docs. |
| src/commands/org_data/integrations/okta_system_integration.ts | Adds central agent to emitted documents and expands PAD-oriented event generation. |
| src/commands/org_data/integrations/entra_id_integration.ts | Adds central agent to sync marker and entity docs. |
| src/commands/org_data/integrations/active_directory_integration.ts | Adds central agent to AD user/computer docs. |
| src/commands/org_data/integrations/cloudtrail_integration.ts | Threads org into event builders and adds central agent to CloudTrail docs. |
| src/commands/org_data/integrations/azure_integration.ts | Threads a shared central agent through Azure log builders and adds agent to emitted docs. |
| src/commands/org_data/integrations/gcp_integration.ts | Adds central agent to GCP audit/firewall docs. |
| src/commands/org_data/integrations/cloud_asset_integration.ts | Adds central agent to asset documents. |
| src/commands/org_data/integrations/cloudflare_logpush_integration.ts | Adds central agent to Cloudflare Logpush http/firewall docs. |
| src/commands/org_data/integrations/google_workspace_integration.ts | Adds central agent to Google Workspace docs. |
| src/commands/org_data/integrations/github_integration.ts | Adds central agent to GitHub audit docs. |
| src/commands/org_data/integrations/gitlab_integration.ts | Adds central agent to GitLab audit/api/auth docs. |
| src/commands/org_data/integrations/slack_integration.ts | Adds central agent to Slack audit docs and threads through helpers. |
| src/commands/org_data/integrations/zoom_integration.ts | Adds central agent to Zoom webhook docs and threads through helper signature. |
| src/commands/org_data/integrations/workday_integration.ts | Adds central agent to Workday docs and threads through helper signature. |
| src/commands/org_data/integrations/servicenow_integration.ts | Adds central agent to incident/change docs and threads through helper signatures. |
| src/commands/org_data/integrations/ti_abusech_integration.ts | Adds central agent to TI docs and threads through helper signatures. |
| src/commands/org_data/integrations/thycotic_ss_integration.ts | Adds central agent to Thycotic Secret Server docs and threads through helper signature. |
| src/commands/org_data/integrations/teleport_integration.ts | Adds central agent to Teleport audit docs and threads through helper signature. |
| src/commands/org_data/integrations/onepassword_integration.ts | Adds central agent to 1Password docs. |
| src/commands/org_data/integrations/o365_integration.ts | Adds central agent to O365 audit docs. |
| src/commands/org_data/integrations/mongodb_atlas_integration.ts | Adds central agent to MongoDB Atlas docs and threads through helper signatures. |
| src/commands/org_data/integrations/mattermost_integration.ts | Adds central agent to Mattermost audit docs and threads through helper signature. |
| src/commands/org_data/integrations/lyve_cloud_integration.ts | Adds central agent to Lyve Cloud audit docs and threads through helper signature. |
| src/commands/org_data/integrations/lastpass_integration.ts | Adds central agent to user/event/shared-folder docs and threads through helper signatures. |
| src/commands/org_data/integrations/keycloak_integration.ts | Adds central agent to Keycloak log docs and threads through helper signature. |
| src/commands/org_data/integrations/keeper_integration.ts | Adds central agent to Keeper audit docs and threads through helper signature. |
| src/commands/org_data/integrations/jumpcloud_integration.ts | Adds central agent to JumpCloud events and threads through helper signature. |
| src/commands/org_data/integrations/hashicorp_vault_integration.ts | Adds central agent to Vault audit/log docs and threads through helper signatures. |
| src/commands/org_data/integrations/forgerock_integration.ts | Adds central agent to ForgeRock docs and threads through helper signatures. |
| src/commands/org_data/integrations/cyberark_pas_integration.ts | Adds central agent to CyberArk PAS audit docs and threads through helper signature. |
| src/commands/org_data/integrations/cisco_duo_integration.ts | Adds central agent to Duo auth docs. |
| src/commands/org_data/integrations/canva_integration.ts | Adds central agent to Canva audit docs and threads through helper signature. |
| src/commands/org_data/integrations/box_integration.ts | Adds central agent to Box event docs and threads through helper signature. |
| src/commands/org_data/integrations/bitwarden_integration.ts | Adds central agent to Bitwarden docs and threads through helper signatures. |
| src/commands/org_data/integrations/beyondinsight_integration.ts | Adds central agent to BeyondInsight docs and threads through helper signatures. |
| src/commands/org_data/integrations/auth0_integration.ts | Adds central agent to Auth0 logs and threads through helper signature. |
| src/commands/org_data/integrations/authentik_integration.ts | Adds central agent to Authentik user/group/event docs and threads through helper signatures. |
| src/commands/org_data/integrations/atlassian_jira_integration.ts | Adds central agent to Jira audit docs and threads through helper signature. |
| src/commands/org_data/integrations/atlassian_confluence_integration.ts | Adds central agent to Confluence audit docs and threads through helper signature. |
| src/commands/org_data/integrations/atlassian_bitbucket_integration.ts | Adds central agent to Bitbucket audit docs and threads through helper signature. |
| src/commands/org_data/integrations/sailpoint_integration.ts | Adds central agent to SailPoint event docs and threads through helper signature. |
| src/commands/org_data/integrations/ping_one_integration.ts | Adds central agent to PingOne audit docs and threads through helper signature. |
| src/commands/org_data/integrations/ping_directory_integration.ts | Adds central agent to PingDirectory SCIM docs and threads through helper signature. |
| .agents/skills/update-org-data-integrations/SKILL.md | Documents the new stable agent fields and correlation rules. |
Comment on lines
+115
to
119
| const hostname = laptop ? `${employee.userName}-${laptop.platform}` : 'unknown'; | ||
| return { | ||
| '@timestamp': timestamp, | ||
| agent: laptop ? this.buildLocalAgent(laptop, hostname) : undefined, | ||
| message: JSON.stringify({ event: rawEvent }), |
There was a problem hiding this comment.
agent is conditionally set to undefined when no laptop is found, but org generation always creates at least one laptop per employee. This conditional adds dead code and can silently omit agent (since undefined fields drop during JSON serialization). Consider removing the fallback and always setting agent via buildLocalAgent(...) to keep documents ECS-consistent.
Comment on lines
+1231
to
+1233
| console.log( | ||
| ` Generating PAD anomalous Okta patterns for ${rogueEmployees.length} rogue actor(s)...`, | ||
| ); |
There was a problem hiding this comment.
Avoid using console.log here; it will spam stdout during generation and bypasses the repo’s structured logger. Use log.info (or a debug-level log) from src/utils/logger.ts, or remove these statements entirely.
Comment on lines
+1294
to
+1296
| // Replace the membership events with properly attributed ones | ||
| events.length = events.length - membershipCount; | ||
| for (let i = 0; i < membershipCount; i++) { |
There was a problem hiding this comment.
This code first pushes membershipCount group membership events and then immediately removes them by truncating the array, only to regenerate the same count with createAdminActionEvent. This wastes work and advances Faker’s RNG unnecessarily. Consider generating only the attributed events (second loop) and deleting the first loop + manual truncation.
Comment on lines
138
to
+142
| private createWebhookDocument( | ||
| employee: Employee, | ||
| org: Organization, | ||
| masterAccountId: string, | ||
| centralAgent: { id: string; name: string; type: string; version: string }, |
There was a problem hiding this comment.
The centralAgent parameter type is duplicated inline here. To avoid drift and keep signatures consistent across integrations, consider importing and using the shared AgentData type from base_integration.ts instead of redefining { id; name; type; version }.
tiansivive
approved these changes
Apr 7, 2026
Resolve conflicts in azure_integration.ts by combining main's improved audit-log logic (distinct target employees, user-user relationship enrichment, always user-initiated) with the branch's centralAgent metadata propagation. Fix duplicate hostname declaration in jamf_pro_integration.ts introduced by the auto-merge. Made-with: Cursor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Updates the organization data model and integration logic to include stable Elastic Agent identification in all generated documents. This ensures that synthetic data correctly mimics real-world ECS-compliant logs and supports security features that rely on agent-based correlation.
It generates per agents per workstation for Endpoint integration and a central agent for SaaS integrations, simulating real world environment.
Key changes:
elasticAgentIdto Device and Host models, andcentralAgentto the Organization model.buildCentralAgent,buildLocalAgent, andbuildServerAgenthelpers inBaseIntegration.agentfield in their documents.Additional changes